Information processing apparatus and information processing method

ABSTRACT

The present disclosure relates to an information processing apparatus and an information processing method for suppressing a decrease in the accuracy of image projection correction. By use of an image projection model using a distortion factor of an fθ lens with an image height of incident light expressed by the product of a focal point distance f and an incident angle θ of the incident light, the posture of a projection section for projecting an image and the posture of an imaging section for capturing a projection plane to which the image is projected are estimated. The present disclosure may be applied, for example, to information processing apparatuses, projection apparatuses, imaging apparatuses, projection imaging apparatuses, projection imaging control apparatuses, or image projection and imaging systems.

TECHNICAL FIELD

The present disclosure relates to an information processing apparatusand an information processing method. More particularly, the disclosurerelates to an information processing apparatus and an informationprocessing method for suppressing a decrease in the accuracy of imageprojection correction.

BACKGROUND ART

Heretofore, there have been projection correction techniques formeasuring three-dimensional shapes by use of a projector-camera system.Some methods representing the techniques involve three-dimensionallymeasuring the shape of a projection plane (screen) and geometricallycorrecting a projected image thereon, based on information regarding themeasurements. In order to measure the three-dimensional shape, it isnecessary to estimate (calibrate) two kinds of parameters: internalparameters (e.g., focal point distance, principal point, and lensdistortion factor) indicative of individual characteristics of theprojector and the camera; and external parameters representative oftheir positions and postures relative to each other.

For example, there have been methods of calibrating beforehand eitherinternal variables or external variables of the projector and camera,the other variables being calibrated on the basis of informationregarding the measurements obtained after configuration of the apparatus(e.g., see PTL 1 and PTL 2).

The above-mentioned projection correction techniques for measuring thethree-dimensional shape by use of the projector-camera system have beenpredicated on the use of what is generally called an f tan θ lens systememployed in an ordinary projector. In the case of projection correctionwith a projector-camera system using a projector having the f tan θlens, the effect of lens distortion in the projector is very smallcompared with the effect caused by the internal and external parameters.For this reason, performing calibration even without regard to the lensdistortion enables projection correction with sufficiently highaccuracy.

CITATION LIST Patent Literature [PTL 1]

Japanese Patent Laid-open No. 2015-142157

[PTL 2]

Japanese Patent Laid-open No. 2005-244835

SUMMARY Technical Problems

On the other hand, what is generally called the fθ lens involves theeffect of lens distortion far larger than the f tan θ lens. Thus, in thecase of projection correction with a projector-camera system using aprojector having the fθ lens, performing calibration without regard tothe lens distortion as in the case of the f tan θ lens system can makethe accuracy of projection correction lower than in the case of the ftan θ lens system.

The present disclosure has been made in view of the above circumstancesand aims at suppressing a decrease in the accuracy of image projectioncorrection.

Solution to Problems

According to one aspect of the present technology, there is provided aninformation processing apparatus including a posture estimation sectionconfigured such that, by use of an image projection model using adistortion factor of an fθ lens with an image height of incident lightexpressed by a product of a focal point distance f and an incident angleθ of the incident light, the posture estimation section estimates aposture of a projection section for projecting an image and a posture ofan imaging section for capturing a projection plane to which the imageis projected.

Also, according to one aspect of the present technology, there isprovided an information processing method including, by use of an imageprojection model using a distortion factor of an fθ lens with an imageheight of incident light expressed by a product of a focal pointdistance f and an incident angle θ of the incident light, estimating aposture of a projection section for projecting an image and a posture ofan imaging section for capturing a projection plane to which the imageis projected.

With the information processing apparatus and the information processingmethod according to one aspect of the present technology, by use of animage projection model using a distortion factor of an fθ lens with animage height of incident light expressed by a product of a focal pointdistance f and an incident angle θ of the incident light, a posture of aprojection section for projecting an image and a posture of an imagingsection for capturing a projection plane to which the image is projectedare estimated.

Advantageous Effects of Invention

According to the present disclosure, it is possible to correct imageprojection. More particularly, the disclosure permits reduction of adecrease in the accuracy of image projection correction.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram depicting a principal configuration example ofa projection imaging system.

FIG. 2 is a block diagram depicting a principal configuration example ofa control apparatus.

FIG. 3 is a functional block diagram depicting examples of majorfunctional blocks implemented by a control section.

FIG. 4 is a block diagram depicting a principal configuration example ofa projection apparatus.

FIG. 5 is a block diagram depicting a principal configuration example ofan imaging apparatus.

FIG. 6 is a flowchart explaining a typical flow of a calibrationprocess.

FIG. 7 is a view depicting how pixel-to-pixel correspondence is obtainedusing structured light.

FIG. 8 is a flowchart explaining a typical flow of a posture estimationprocess.

FIG. 9 is a flowchart explaining a typical flow of a parameterestimation process.

FIG. 10 is a view depicting how distortion is typically corrected.

FIG. 11 is a view depicting how a ray trace is typically performed withdistortion taken into consideration.

FIG. 12 is a view depicting how posture estimation is typicallyperformed.

FIG. 13 is a flowchart explaining a typical flow of a geometriccorrection process.

FIG. 14 is a view depicting how geometric projection correction istypically performed with respect to a virtual viewpoint.

FIG. 15 is a view depicting how a virtual viewpoint is typically set.

FIG. 16 is a view depicting how two-dimensional curved surface fittingis typically performed.

FIG. 17 is a view depicting how model misalignment typically takesplace.

FIG. 18 is a view depicting how a model misalignment correspondingproc251ess is typically performed.

FIG. 19 is a block diagram depicting another configuration example ofthe projection imaging system.

DESCRIPTION OF EMBODIMENTS

The modes for implementing the present disclosure (hereinafter referredto as the embodiments) are described below. Note that the descriptionwill be given in the following order.

1. Image projection correction

2. First embodiment (projection imaging system using the fθ lens)

3. Second embodiment (another configuration example of the projectionimaging system)

4. Notes

1. Image Projection Correction

<Projection Correction of the f Tan θ Lens System>

In order to project a single image to a screen by using multipleprojectors, needed are techniques taking into consideration thedistortion of individual images from the projectors on a curved surfaceof the screen and correcting the projected images accordingly into ageometrically accurate, distortion-free image. Some methods representingsuch techniques involve getting the configured projectors to projectpatterns or markers to the screen and causing the cameras or sensors,which are also configured, to obtain information for correctionpurposes.

For example, two kinds of methods have been proposed as projectioncorrection technique using cameras in non-planar projection: a methodusing two-dimensional information and based on the appearance ofcaptured images for performing correction; and a method usingthree-dimensional information and involving estimation (=calibration) ofboth the internal variables indicative of the characteristics of thecameras and projectors and the external variables representative oftheir relative positions and postures for correction based on projectionranges of the projectors, i.e., information regarding the measurementsof the screen shape.

The method using two-dimensional information involves a simplifiedapparatus configuration without the need for calibrating the projectorsor cameras. However, this method does not guarantee that the correctedimage is geometrically accurate (e.g., a straight line when correctedought to be seen as a straight line from a camera point of view). Bycontrast, the method using three-dimensional information, with itscorrection aligned with the screen shape, is more likely to guaranteethe geometric accuracy of the resulting image but requires the followingprocedures for calibration.

That is, with the method using three-dimensional information, theprojector projects patterns or makers to a target; the camera capturesthe projected patterns or markers; and the control apparatus obtainspixel-to-pixel correspondence between the projector and the camera byusing the captured image and measures depth information (depth) by usingthe principle or triangulation. At this time, in measuring the depthfrom the pixel-to-pixel correspondence between the projector and thecamera, the control apparatus is required to follow procedures forestimating the internal variables of the projector and camera and theirrelative positions and postures, i.e., the control apparatus requirescalibration in the case of the method using three-dimensionalinformation.

The above requirements are met, for example, by calibrating all internaland external variables of each of the projector and camera enclosuresbefore configuration of the apparatus. Alternatively, as described inthe above-cited PTL 1 and PTL 2, either the internal variables or theexternal variables of each projector and camera enclosure are calibratedbeforehand, with the other variables being calibrated on the basis ofthe measurement information after configuration of the apparatus.

However, in a case where the projectors or the cameras have been movedor where the internal variables have been changed typically byzoom/shift operations, the above method requires tedious recalibrationwork. Where a single image is projected by multiple projectors, with theprojections left uncorrected, there occurs image misalignment in anoverlap region between the projections by the multiple projectors. Themisalignment is incurred even by slight movement of a projector afterthe projection correction. This requires subsequent tedious,time-consuming work on preliminary procedures for recalibration, whichis not very practical in an actual operation setup.

Further, the existing calibration methods are based on the assumptionthat the methods are applied to a system using a projector equipped withwhat is generally called the f tan θ lens (ordinary lens) of which theimage height of light at an incident angle θ is represented by theproduct of a focal point distance f and tan θ (f·tan θ). In the case ofprojection correction with a projector-camera system using a projectorhaving the f tan θ lens, the effect of lens distortion in the projectoris very small compared with the effect from the internal and externalparameters. Thus, performing calibration even without regard to the lensdistortion enables projection correction with sufficiently highaccuracy.

On the other hand, in the case of recently introduced image projectionsystems by which, for example, images are projected to the projectionplane of a dome-type curved surface (e.g., semispherical shape) toenhance the sense of immersion of a user watching the projected images,it has been proposed to use projectors furnished with what is generallycalled the fθ lens (also known as the fish-eye lens) of which the imageheight of incident light is represented by the product of the focalpoint distance f and the incident angle θ of the incident light.

Using the fθ lens permits a wider angle of view of the projected imagethan when the f tan θ lens is used. Thus, the projector using the fθlens is more suitable for image projection onto the curved surface suchas the dome-type projection plane than the projector employing the f tanθ lens.

However, what is generally called the fθ lens is subject to a far highereffect of lens distortion than the f tan θ lens. Thus in the case ofprojection correction with a projector-camera system using a projectorhaving the fθ lens, performing calibration without regard to the lensdistortion as in the case of the f tan θ lens system can make theaccuracy of projection correction lower than in the case of the f tan θlens system.

Whereas the above-cited PTL 1 discloses a projector-camera system usinga fish-eye projector, the method described therein involves projectingimages to two projection planes, i.e., a plane in the optical axisdirection of the projector, and a plane perpendicular to that plane. Forthis reason, it is difficult to apply this method to image projectiononto the dome-type spherical surface screen.

Thus, for image projection correction, an image projection model thatuses the distortion factor of the fθ lens with the image height ofincident light represented by the product of the focal point distance fand the incident angle θ of the incident light is used to estimate theposture of a projection section for projecting an image and the postureof an imaging section for capturing the projection plane to which theimage is projected.

For example, an information processing apparatus includes a postureestimation section for estimating the posture of a projection sectionthat projects an image and the posture of an imaging section thatcaptures the projection plane to which the image is projected, throughthe use of an image projection model that uses the distortion factor ofthe fθ lens with the image height of incident light represented by theproduct of the focal point distance f and the incident angle θ of theincident light.

The above configuration permits posture estimation while correcting thelens distortion of the fθ lens. Thus, even in the case where the fθ lensis used, it is possible to suppress a decrease in the accuracy of imageprojection correction. In other words, the above configuration enablescalibration of the internal and external parameters of the projectionsection and imaging section with sufficiently high accuracy. That is,image projection correction is easier to perform, so that the robustnessof the accuracy of parameters in the face of environmental changes isenhanced. This enables practical operation of the image projectionsystem that uses the fθ lens.

2. First Embodiment

<Projection Imaging System>

FIG. 1 is a block diagram depicting a principal configuration example ofa projection imaging system to which the present technology is applied.In FIG. 1, a projection imaging system 100 is a system that projectsimages to a projection plane and calibrates parameters by using imagescaptured of the images projected onto the projection plane.

As depicted in FIG. 1, the projection imaging system 100 includes acontrol apparatus 111, a projection apparatus 112-1, an imagingapparatus 113-1, a projection apparatus 112-2, and an imaging apparatus113-2. The projection apparatus 112-1, the imaging apparatus 113-1, theprojection apparatus 112-2, and the imaging apparatus 113-2 arecommunicably connected with the control apparatus 111 via cables 115-1to 115-4, respectively.

In the description that follows, the projection apparatuses 112-1 and112-2 will be referred to as the projection apparatus or apparatuses 112in the case where there is no need for their individual explanation.Also, the imaging apparatuses 113-1 and 113-2 will be referred to as theimaging apparatus or apparatuses 113 where there is no need for theirindividual explanation. Further, the cables 115-1 to 115-4 will bereferred to as the cable or cables 115 where there is no need for theirindividual explanation.

The control apparatus 111 controls each projection apparatus 112 andeach imaging apparatus 113 via the cables 115. For example, the controlapparatus 111 is supplied with an image via a cable 114. The controlapparatus 111 feeds the image to each projection apparatus 112 that inturn projects the image to a dome-type (partially sphericalsurface-shaped) screen 121. As another example, the control apparatus111 causes each imaging apparatus 113 to capture the screen 121 (e.g.,image projected onto the screen 121) and acquires the captured image.

As a further example, the control apparatus 111 calibrates theparameters of the projection apparatuses 112 and imaging apparatuses 113by using the captured image, thereby calculating the parameters forgeometrically correcting the images to be projected by the projectionapparatuses 112. Using the calculated parameters, the control apparatus111 geometrically corrects images supplied from the outside and feedsthe geometrically corrected images to the projection apparatuses 112.

The projection apparatuses 112 each have the function of what isgenerally called a projector. For example, under the control of thecontrol apparatus 111, the projection apparatuses 112 project to thescreen 121 images supplied from the control apparatus 111. Theprojection apparatuses 112 under the control of the control apparatus111 operate in cooperation with each other to perform image projectionsuch that a single projected image appears on the screen 121 (i.e., oneprojected image is displayed on the screen 121).

For example, the multiple projection apparatuses 112 perform imageprojection in such a manner that the images projected are arranged sideby side with no gap therebetween on the screen 121, thereby obtaining aprojected image larger (with high resolution) than the image projectedby a single projection apparatus 112 (i.e., such a projected image isdisplayed on the screen 121). As another example, the multipleprojection apparatuses 112 perform image projection in such a mannerthat the images projected coincide with each other in position on thescreen 121, thereby acquiring an image brighter (of high dynamic range)than the image projected by a single projection apparatus 112 (i.e.,such a projected image is displayed on the screen 121). That is, theprojection imaging system 100 in such a case is what is generally calleda multi-projection system that implements what is known as projectionmapping.

The imaging apparatuses 113 each have the function of what is generallycalled a camera. For example, under the control of the control apparatus111, the imaging apparatuses 113 capture the screen 121 (i.e., screen121 to which images are projected by the projection apparatuses 112) andfeeds data of the captured images (also called captured image data) tothe control apparatus 111. The captured images are used by the controlapparatus 111 in calculating the parameters for geometrically correctingimages (i.e., in calibrating the parameters of the projectionapparatuses 112 and imaging apparatuses 113). That is, the imagingapparatuses 113 are configured to geometrically correct the images to beprojected (i.e., configured to calculate the parameters for geometriccorrection).

The screen 121 is an approximately dome-shaped (partially sphericalsurface-shaped) projection plane. Configured to be a curved surface, thescreen 121 allows images to be projected (displayed) thereon with awider viewing angle than when the images are projected onto a flatscreen. This enables the user to experience more realistic sensationsand a deeper sense of immersion.

Further, the projection apparatuses 112 and the imaging apparatuses 113each include what is generally called the fθ lens (also known as thefish-eye lens) instead of what is generally called the f tan θ lens(ordinary lens). It follows that the images projected by the projectionapparatuses 112 or captured by the imaging apparatuses 113 each havelarger distortion, particularly in a peripheral region, than in the caseof the f tan θ lens.

<Control Apparatus>

FIG. 2 is a block diagram depicting a principal configuration example ofthe control apparatus 111 as an embodiment of the information processingapparatus to which the present technology is applied. It is to be notedthat FIG. 2 depicts major processing blocks and principal data flowstherebetween and does not cover the entire configuration of the controlapparatus 111. That is, the control apparatus 111 may include processingblocks that are not illustrated in FIG. 11 as well as data flows andprocesses other than those indicated by arrows in FIG. 2.

As depicted in FIG. 2, the control apparatus 111 includes a controlsection 201, an input section 211, an output section 212, a storagesection 213, a communication section 214, and a drive 215.

The control section 201 performs processes related to controls. Forexample, the control section 201 controls any configured elements in thecontrol apparatus 111. The control section 201 also performs processesrelated to controls over other apparatuses such as the projectionapparatuses 112 and imaging apparatuses 113. The control section 201 maybe configured in any manner desired. For example, the control section201 may include a CPU (Central Processing Unit), a ROM (Read OnlyMemory), and RAM (Random Access Memory), the CPU loading programs anddata from the ROM into the RAM and executing and operating on the loadedprograms and data to carry out relevant processes.

The input section 211 includes input devices for accepting informationfrom the outside such as the input from the user. For example, the inputsection 211 may include a keyboard, a mouse, operation buttons, a touchpanel, a camera, a microphone, and input terminals. The input section211 may further include various sensors such as an acceleration sensor,an optical sensor, and a temperature sensor, as well as input equipmentsuch as a barcode reader. The output section 212 includes output devicesfor outputting information such as images and sounds. For example, theoutput section 212 may include a display unit, speakers, and outputterminals.

The storage section 213 includes storage media for storing informationsuch as programs and data. For example, the storage section 213 mayinclude a hard disk, a RAM disk, and a nonvolatile memory. Thecommunication section 214 includes a communication device forcommunicating with external apparatuses by sending and receivinginformation such as programs and data thereto and therefrom viapredetermined communication media (e.g., suitable networks such as theInternet). For example, the communication section 214 may include anetwork interface. The communication section 214 performs communication(i.e., exchanges programs and data), for example, with apparatusesexternal to the control apparatus 111 (e.g., projection apparatuses 112and imaging apparatuses 113). Preferably, the communication section 214may have a wired communication function or a wireless communicationfunction, or both.

The drive 215 retrieves information (e.g., programs and data) fromremovable media 221 such as a magnetic disk, an optical disk, amagneto-optical disk, or a semiconductor memory attached to the drive215. The drive 215 supplies the information retrieved from the removablemedia 221 to the control section 201, among others. In the case where arewritable piece of the removable media 221 is attached to the drive215, the drive 215 can have the information (e.g., programs and data)supplied from the control section 201 stored in the attached piece ofremovable media 221.

<Functional Blocks of the Control Apparatus>

FIG. 3 is a functional block diagram depicting examples of majorfunctional blocks implemented by the control apparatus 111 performingprograms, for example. As depicted in FIG. 3, the control apparatus 111executes programs to implement the functions of a sensing processingsection 251, a posture estimation section 252, and a geometriccorrection section 253, for example.

The sensing processing section 251 performs processes related tosensing. For example, the sensing processing section 251 performs theprocess of detecting corresponding points between the pixels of theprojection apparatus 112 and those of the imaging apparatus 113 by usingcaptured images from the imaging apparatuses 113. The sensing processingsection 251 supplies the posture estimation section 252 with the resultof the process (i.e., information regarding the corresponding pointsbetween the projection apparatus 112 and the imaging apparatus 113).

The posture estimation section 252 performs processes related toestimation of the postures of the projection apparatuses 112 and imagingapparatuses 113. For example, using an image projection model that usesthe distortion factor of the fθ lens, the posture estimation section 252estimates the parameters (variables) related to the postures of at leasteither the projection apparatuses 112 or the imaging apparatuses 113(i.e., calculates the estimates of the variables related to postures).

The posture-related parameters may be of any suitable type. For example,the parameters may include the internal parameters (also called internalvariables) of at least either the projection apparatuses 112 or theimaging apparatuses 113. The internal parameters may be of any suitabletype. For example, the internal parameters may include at least one ofthe focal point distance, principal point, or the parameter (k_(inv))corresponding to inverse transformation of the lens distortion factor ofthe projection apparatus 112 or the imaging apparatus 113.

Alternatively, the posture-related parameters may include the externalparameters (also called external variables) of at least either theprojection apparatuses 112 or the imaging apparatuses 113. The externalparameters may be of any suitable type. For example, the externalparameters may include at least either a rotation matrix or atranslation vector with respect to the origin of a world coordinatesystem of the projection apparatus 112 or of the imaging apparatus 113.

The posture estimation section 252 estimates the posture-relatedparameters, based on the information regarding the corresponding pointsbetween the projection apparatus 112 and the imaging apparatus 113, theinformation being supplied from the sensing processing section 251. Theposture estimation section 252 also estimates the posture-relatedparameters, based on representative values of the internal parameters(also called internal variable representative values) of at least eitherthe projection apparatuses 112 or the imaging apparatuses 113 havingbeen determined beforehand.

The posture estimation section 252 supplies the geometric correctionsection 253 with the obtained parameter estimates (at least either theinternal parameter estimates (also called internal variable estimates)or the external parameter estimates (also called external variableestimates) of at least either the projection apparatuses 112 or theimaging apparatuses 113).

The geometric correction section 253 performs processes related togeometric correction of images. For example, on the basis of theparameter estimates supplied from the posture estimation section 252,the geometric correction section 253 calculates the parameters (e.g.,vector data for geometric correction) for use in geometric correction ofthe images input from the outside via the cable 114.

<Posture Estimation Section>

As depicted in FIG. 3, the posture estimation section 252 includes animaging variable estimation section 261, a projection variableestimation section 262, and a total optimization section 263.

The imaging variable estimation section 261 performs processes relatedto estimating at least either the internal parameters or the externalparameters of the imaging apparatuses 113 (the parameters are alsocalled imaging variables). The projection variable estimation section262 performs processes related to estimating at least either theinternal parameters or the external parameters of the projectionapparatuses 112 (the parameters are also called projection variables).The total optimization section 263 performs processes related tooptimizing the estimates of the imaging variables (also called imagingvariable estimates) obtained by the imaging variable estimation section261 and the estimates of the projection variables (also called theprojection variable estimates) acquired by the projection variableestimation section 262.

In this manner, the posture estimation section 252 estimates the imagingvariables and projection variables and, through total optimization,obtains at least either the internal variable estimates or the externalvariable estimates of at least either the projection apparatuses 112 orthe imaging apparatuses 113.

At this time, the posture estimation section 252 performs theabove-described posture estimation by using the image projection modelthat uses the distortion factor of the fθ lens. That is, the imagingvariable estimation section 261 obtains the imaging variable estimatesby using the image projection model that uses the distortion factor ofthe fθ lens. Likewise, the projection variable estimation section 262acquires the projection variable estimates by use of the imageprojection model that uses the distortion factor of the fθ lens.Similarly, the total optimization section 263 optimizes all of theseparameters through the use of the image projection model that uses thedistortion factor of the fθ lens.

Thus, even in the case where the projection apparatuses 112 or theimaging apparatuses 113 use the fθ lens, the control apparatus 111 cansuppress a decrease in the accuracy of image projection correction. Thispermits practical operation of the projection imaging system 100 usingthe fθ lens.

<Geometric Correction Section>

Further, as depicted in FIG. 3, the geometric correction section 253includes a projection plane modeling section 271, a virtual viewpointposition/projection direction estimation section 272, a modelmisalignment corresponding processing section 273, and a projection maskgeneration section 274.

The projection plane modeling section 271 performs processes related toprojection plane modeling (functionalization of curved surface). Thevirtual viewpoint position/projection direction estimation section 272performs processes related to estimating a virtual viewpoint positionserving as a reference point for distortion correction and an imageprojection direction relative to that virtual viewpoint position. Themodel misalignment corresponding processing section 273 performs acorresponding process for suppressing misalignment between the actualprojection plane and the model thereof (also called model misalignment).The projection mask generation section 274 performs processes related togenerating projection masks for limiting the range in which theprojection apparatuses 112 project images.

<Projection Apparatuses>

FIG. 4 is a block diagram depicting a principal configuration example ofthe projection apparatus 112 as one embodiment of the informationprocessing apparatus to which the present technology is applied. It isto be noted that FIG. 4 depicts major processing blocks and principaldata flows therebetween and does not cover the entire configuration ofthe projection apparatus 112. That is, the projection apparatus 112 mayinclude processing blocks that are not illustrated in FIG. 4 as well asdata flows and processes other than those indicated by arrows in FIG. 4.

As depicted in FIG. 4, the projection apparatus 112 includes a controlsection 301, a projection section 302, an input section 311, an outputsection 312, a storage section 313, a communication section 314, and adrive 315.

The control section 301 performs processes related to controls. Forexample, the control section 301 controls any configured elements in theprojection apparatus 112. For example, the control section 301 controlsdrive of the projection section 302. The control section 301 may beconfigured in any manner desired. For example, the control section 301may include a CPU, a ROM, and RAM, the CPU loading programs and datafrom the ROM into the RAM and executing and operating on the loadedprograms and data to carry out relevant processes.

The projection section 302 under the control of the control section 301performs processes related to image projection. For example, theprojection section 302 acquires from the control section 301 the imagedata supplied from the control apparatus 111 and projects the acquiredimage to the screen 121. The projection section 302 has the fθ lens asmentioned above, so that the image is projected to the screen 121 viathe fθ lens.

The input section 311 includes input devices for accepting informationfrom the outside such as the input from the user. For example, the inputsection 311 may include a keyboard, a mouse, operation buttons, a touchpanel, a camera, a microphone, and input terminals. The input section311 may further include various sensors such as an acceleration sensor,an optical sensor, and a temperature sensor, as well as input equipmentsuch as a barcode reader. The output section 312 includes output devicesfor outputting information such as images and sounds. For example, theoutput section 312 may include a display unit, speakers, and outputterminals.

The storage section 313 includes storage media for storing informationsuch as programs and data. For example, the storage section 313 mayinclude a hard disk, a RAM disk, and a nonvolatile memory. Thecommunication section 314 includes a communication device forcommunicating with external apparatuses by sending and receivinginformation such as programs and data thereto and therefrom viapredetermined communication media (e.g., suitable networks such as theInternet). For example, the communication section 314 may include anetwork interface. The communication section 314 performs communication(i.e., exchanges programs and data), for example, with apparatusesexternal to the projection apparatus 112 (e.g., control apparatus 111).Preferably, the communication section 314 may have a wired communicationfunction or a wireless communication function, or both.

The drive 315 retrieves information (e.g., programs and data) fromremovable media 321 such as a magnetic disk, an optical disk, amagneto-optical disk, or a semiconductor memory attached to the drive315. The drive 315 supplies the information retrieved from the removablemedia 321 to the control section 301, among others. In the case where arewritable piece of the removable media 321 is attached to the drive315, the drive 315 can have the information (e.g., programs and data)supplied from the control section 301 stored in the attached piece ofremovable media 321.

<Imaging Apparatuses>

FIG. 5 is a block diagram depicting a principal configuration example ofthe imaging apparatus 113 as one embodiment of the informationprocessing apparatus to which the present technology is applied. It isto be noted that FIG. 5 depicts major processing blocks and principaldata flows therebetween and does not cover the entire configuration ofthe imaging apparatus 113. That is, the imaging apparatus 113 mayinclude processing blocks that are not illustrated in FIG. 5 as well asdata flows and processes other than those indicated by arrows in FIG. 5.

As depicted in FIG. 5, the imaging apparatus 113 includes a controlsection 401, an imaging section 402, an input section 411, an outputsection 412, a storage section 413, a communication section 414, and adrive 415.

The control section 401 performs processes related to controls. Forexample, the control section 401 controls any configured elements in theimaging apparatus 113. For example, the control section 401 controlsdrive of the imaging section 402. The control section 401 may beconfigured in any manner desired. For example, the control section 401may include a CPU, a ROM, and RAM, the CPU loading programs and datafrom the ROM into the RAM and executing and operating on the loadedprograms and data to carry out relevant processes.

The imaging section 402 under the control of the control section 401performs processes related to capturing an imaged subject. For example,the imaging section 402 captures the image projected to the screen 121by the projection apparatus 112 so as to obtain captured image data. Theimaging section 402 supplies the captured image data to the controlsection 401. In turn, the control section 401 supplies the capturedimage data to the control apparatus 111 via the communication section414. Note that the imaging section 402 has the fθ lens as mentionedabove, so that the imaging section 402 captures the screen 121 (i.e.,projected image) via the fθ lens.

The input section 411 includes input devices for accepting informationfrom the outside such as the input from the user. For example, the inputsection 411 may include a keyboard, a mouse, operation buttons, a touchpanel, a camera, a microphone, and input terminals. The input section411 may further include various sensors such as an acceleration sensor,an optical sensor, and a temperature sensor, as well as input equipmentsuch as a barcode reader. The output section 412 includes output devicesfor outputting information such as images and sounds. For example, theoutput section 412 may include a display unit, speakers, and outputterminals.

The storage section 413 includes storage media for storing informationsuch as programs and data. For example, the storage section 413 mayinclude a hard disk, a RAM disk, and a nonvolatile memory. Thecommunication section 414 includes a communication device forcommunicating with external apparatuses by sending and receivinginformation such as programs and data thereto and therefrom viapredetermined communication media (e.g., suitable networks such as theInternet). For example, the communication section 414 may include anetwork interface. The communication section 414 performs communication(i.e., exchanges programs and data), for example, with apparatusesexternal to the imaging apparatus 113 (e.g., control apparatus 111).Preferably, the communication section 414 may have a wired communicationfunction or a wireless communication function, or both.

The drive 415 retrieves information (e.g., programs and data) fromremovable media 421 such as a magnetic disk, an optical disk, amagneto-optical disk, or a semiconductor memory attached to the drive415. The drive 415 supplies the information retrieved from the removablemedia 421 to the control section 401, among others. In the case where arewritable piece of the removable media 421 is attached to the drive415, the drive 415 can have the information (e.g., programs and data)supplied from the control section 401 stored in the attached piece ofremovable media 421.

<Flow of the Calibration Process>

The processing performed by the above-described projection imagingsystem 100 is explained below. The control apparatus 111 in theprojection imaging system 100 performs a calibration process tocalibrate the projection variables (internal and external parameters ofthe projection apparatuses 112) and the imaging variables (internal andexternal parameters of the imaging apparatuses 113).

A typical flow of the calibration process is explained below withreference to the flowchart of FIG. 6. When the calibration process isstarted, the sensing processing section 251 performs a sensing processin step S101 to detect corresponding points.

In the sensing process, as depicted in the example of FIG. 7, theStructured Light method is used to obtain pixel-to-pixel correspondencebetween the projection apparatus 112 (e.g., projector) and the imagingapparatus 113 (e.g., camera). More specifically, the projectionapparatus 112 (projector) projects to the dome-type screen 121 patternswith their pixels encoded in the time direction (e.g., gray code orchecker pattern) while switching the patterns in time series. Further,the imaging apparatus 113 (camera) captures a projected image of each ofthese patterns. On the basis of each of the patterns included in thecaptured images, the control apparatus 111 obtains the correspondingpoints between the pixels of the projection apparatus 112 and those ofthe imaging apparatus 113. When the information regarding thecorresponding points (i.e., corresponding points between the projectionand imaging apparatuses) has been obtained, the process advances to stepS102.

In step S102, the posture estimation section 252 performs a postureestimation process based on the information regarding the correspondingpoints obtained in step S101, so as to obtain the internal and externalvariable estimates of each of the projection apparatuses 112 and imagingapparatuses 113.

Note that posture estimation section 252 initially regards the internaland external variables of the projection apparatuses 112 and imagingapparatuses 113 as unknowns. After the projection apparatuses 112 andimaging apparatuses 113 have been suitably arranged relative to thescreen 121 to which the projection apparatuses 112 project images, theposture estimation section 252 estimates the variables according to thisarrangement. That is, with the present method, there is no need toperform preliminary calibration procedures on the internal and externalvariables of the projection apparatuses 112 and imaging apparatuses 113.

However, the projection apparatuses 112 and imaging apparatuses 113 maypreferably retain as initial values the representative values of theirinternal variables (e.g., focal point distance, principal point, andlens distortion). The posture estimation section 252 may then performthe posture estimation process using these representative values. Forexample, the focal point distance and the principal point may be set onthe basis of the resolution of captured and projected images. An averageof the values obtained by calibrating multiple projection apparatuses112 and imaging apparatuses 113 beforehand may be used as the lensdistortion factor. These internal variables are used solely as theinitial values. In the processing of step S102 (posture estimationprocess), the posture estimation section 252 estimates again all theseinternal variables. Thus, even in the case where the projectionapparatuses 112 and imaging apparatuses 113 are configured withprojection and imaging apparatuses not used for obtaining therepresentative values of the internal variables, performing the postureestimation process makes it possible to suppress a decrease in theaccuracy of image projection correction.

Note that, by carrying out the posture estimation process, the postureestimation section 252 further estimates the external variables of theprojection apparatuses 112 and imaging apparatuses 113. Obtaining theexternal variables does not require preparing their initial valuesbeforehand. It is possible to automatically estimate the externalvariables in a state where they are completely unknown.

After the internal and external variable estimates of each of theprojection apparatuses 112 and imaging apparatuses 113 have beenacquired by the above-described posture estimation process, the processadvances to step S103.

In step S103, the geometric correction section 253 obtains vector datafor geometric correction by performing a geometric correction processusing the internal and external variable estimates of each of theprojection apparatuses 112 and imaging apparatuses 113 acquired in theprocessing of step S102.

Upon completion of the processing in step S103, the calibration processis terminated.

<Flow of the Posture Estimation Process>

A typical flow of the posture estimation process performed in step S102of FIG. 6 is explained below with reference to the flowchart of FIG. 8.

When the posture estimation process is started, the imaging variableestimation section 261 in the posture estimation section 252 estimatesthe internal and external variables of the imaging apparatuses 113(i.e., imaging variables) in step S121. As mentioned above, it is notnecessary to estimate the external variables beforehand because they canbe estimated by this process. The representative values of the internalvariables are used as their initial values. At this time, the imagingvariable estimation section 261 estimates the posture-related parametersof the imaging apparatuses 113 (i.e., imaging variables) by using theimage projection model that uses the distortion factor of the fθ lens.With the imaging variables estimated, the process advances to step S122.

In step S122, the projection variable estimation section 262 estimatesthe internal and external variables of the projection apparatuses 112(i.e., projection variables). The projection variables are estimated ina manner similar to the case where the imaging variables are estimatedin step S121.

In step S123, the total optimization section 263 optimizes the estimatesof the imaging variables obtained in step S121 (internal and externalvariable estimates of the imaging apparatuses 113) and the estimates ofthe projection variables acquired in step S122 (internal and externalvariable estimates of the projection apparatuses 112).

After each of the variable estimates is optimized and after theprocessing of step S123 is terminated, the posture estimation processcomes to an end. The process then returns to the flowchart of FIG. 6.

As described above, the posture estimation section 252 performs theposture estimation process to individually estimate and optimize theinternal parameters (focal point distance, principal point, andparameter k_(inv) corresponding to inverse transformation of the lensdistortion factor) and the external parameters (rotation matrix andtranslation vector with respect to the origin of a world coordinatesystem) of the projection apparatuses 112, before finally andsimultaneously optimizing the parameters to obtain the final estimates.

<Image Projection Model>

Explained next is the estimation of the imaging variables (step S121) aswell as the estimation of the projection variables (step S122) performedduring the above-described posture estimation process. The imagingvariables and the projection variables are estimated basically usingsimilar methods. These variables are estimated using the imageprojection model that uses the fθ lens.

The definition of the above model is explained first. When athree-dimensional point P in a three-dimensional space has a coordinatevalue X in a world coordinate system and when the point P is transformedto a camera coordinate system, the coordinate value X is transformed toa coordinate value X_(c) as expressed by the following mathematicalexpression (1), by using a rotation matrix R and a translation vector Tfor transformation from the world coordinate system to the cameracoordinate system.

[Math 1]

X _(c) =RX+T  (1)

where, x=X_(c1), y=X_(c2), Z=X_(c3)

When the point P is projected to a plane z=1 as a perspective projectionmodel, the point P is given homogeneous coordinates a,b from which anangle of view θ is obtained as defined by the following expressions (2)to (5).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack & \; \\{a = \frac{x}{z}} & (2) \\\left\lbrack {{Math}.\mspace{14mu} 3} \right\rbrack & \; \\{b = \frac{y}{z}} & (3) \\\left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack & \; \\{r^{2} = {a^{2} + b^{2}}} & (4) \\\left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack & \; \\{\theta = {\arctan (r)}} & (5)\end{matrix}$

Here, with the distortion factor k of the fish-eye lens defined as inthe following expression (6), homogeneous coordinates x′,y′ inconsideration of the distortion are calculated as expressed by thefollowing expressions (7) to (9).

[Math. 6]

k=[k ₁ k ₂ k ₃ k ₄]^(T)  (6)

[Math. 7]

θ_(d)=θ(1+k ₁θ² +k ₂θ⁴ +k ₃θ⁶ +k ₄θ⁸)  (7)

[Math. 8]

x′=(θ_(d) /r)a  (8)

[Math. 9]

y′=(θ_(d) /r)b  (9)

The above term (θ_(d)/r) represents misalignment in image coordinatesdue to the lens distortion and, when k=0, corresponds to an idealfish-eye lens model (r=f θ). Finally, the coordinates x′,y′ that becomeimage coordinates u,v when transformed to the camera coordinate systemare calculated using focal point distances fx,fy as expressed by theexpressions (10) and (11) below. Note that in the expressions, the termscx,cy denote the principal points of the camera, and the value α is aparameter indicative of a shear factor, with α=0 this time.

[Math. 10]

u=f _(x)(x′+αy′)+c _(x)  (10)

[Math. 11]

v=f _(y) y′+c _(y)  (11)

In this case, the internal parameters (e.g., focal point distance andprincipal point) of the projection apparatuses 112 and imagingapparatuses 113 may be expressed by the expression (12) below. In thecase where the scale in the depth direction is assumed to be 1, theinternal parameters may be expressed by the expression (13) below. Thus,when the parameters are multiplied by A or by A⁻¹, mutual transformationbetween two-dimensional image coordinates and three-dimensionalcoordinates is made possible.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 12} \right\rbrack & \; \\{A = \begin{bmatrix}f_{x} & 0 & c_{x} \\0 & f_{y} & c_{y} \\0 & 0 & 1\end{bmatrix}} & (12) \\\left\lbrack {{Math}.\mspace{14mu} 13} \right\rbrack & \; \\{\begin{bmatrix}u \\v \\1\end{bmatrix} = {\begin{bmatrix}f_{x} & 0 & c_{x} \\0 & f_{y} & c_{y} \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}x^{\prime} \\y^{\prime} \\1\end{bmatrix}}} & (13)\end{matrix}$

Using this model makes it possible to implement the process ofre-projecting points in a three-dimensional space first onto an imageplane before transforming the points to the image coordinates that takeinto consideration the lens distortion in the case of using the fish-eyelens (fθ lens).

<Parameter Corresponding to Inverse Transformation of the LensDistortion Factor>

In order to perform a ray trace using the above-described model, thepresent method uses the parameter k_(inv) corresponding to inversetransformation of the lens distortion factor. With this method, it isassumed that a three-dimensional space in which the target to bemeasured exists is an ideal space free of distortion and that the pixelvalue expressed in the image coordinate system of the projectionapparatuses 112 (or imaging apparatuses 113) and corresponding to apoint in a three-dimensional space includes distortion. At this time, apixel p′ in the image coordinate system of the projection apparatuses112 (or imaging apparatuses 113) is subjected to distortion correctionto obtain an ideal coordinate value p (free of distortion), which inturn is projected to a three-dimensional space to acquire a light ray onwhich exists a three-dimensional point P corresponding to the point p′.Thus, on the basis of the pixel-to-pixel correspondence between theprojection apparatus 112 and the imaging apparatus 113, correspondingideal coordinates are projected to the three-dimensional space to obtainmultiple light rays of which the intersection point is measured as athree-dimensional point corresponding to each pixel.

With the above-described model, the value k is defined as the parameterfor re-projecting a point from the three-dimensional space in thedirection of a distorted two-dimensional image. Usually, projecting apoint in the direction of the three-dimensional space through distortioncorrection requires repeated compensation of the distortion of eachpixel. By contrast, the present method introduces the parameter k_(inv)corresponding to inverse transformation of the lens distortion factor kin order to perform distortion correction on the coordinate values ofthe projection apparatuses 112 and imaging apparatuses 113 by use of theabove-described expressions (7) to (9). Using the parameter k_(inv)permits unified distortion correction on all pixels. Further, comparedwith methods of compensating each pixel, this method suppresses anincrease in calculation costs. The parameter kin, is initially estimatedby the procedure below using the value of the distortion factor k.Thereafter, the parameter k_(inv) is estimated again in optimizationsteps (parameter estimation process), to be discussed later.

The mathematical expression (7) above is arranged with respect to thedistortion factor k and expressed in matrix form as defined by thefollowing expression (14).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 14} \right\rbrack & \; \\{{\theta_{d} - \theta} = {\begin{bmatrix}\theta^{3} & \theta^{5} & \theta^{7} & \theta^{9}\end{bmatrix}\begin{bmatrix}k_{1} \\k_{2} \\k_{3} \\k_{4}\end{bmatrix}}} & (14)\end{matrix}$

The expression (14) above is obtained for each angle of view elcorresponding to one pixel in the images of the projection apparatuses112 and imaging apparatuses 113. Thus, the process of lens distortiontransformation on “n” pixels is expressed by multiplication of an (n×4)matrix with a (4×1) matrix as defined by the following expression (15).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 15} \right\rbrack & \; \\{\begin{bmatrix}{\theta_{d\; 1} - \theta_{1}} \\{\theta_{d\; 2} - \theta_{2}} \\\vdots \\{\theta_{dn} - \theta_{n}}\end{bmatrix} = {\begin{bmatrix}\theta_{1}^{3} & \theta_{1}^{5} & \theta_{1}^{7} & \theta_{1}^{9} \\\theta_{2}^{3} & \theta_{2}^{5} & \theta_{2}^{7} & \theta_{2}^{9} \\\vdots & \vdots & \vdots & \vdots \\\theta_{n}^{3} & \theta_{n}^{5} & \theta_{n}^{7} & \theta_{n}^{9}\end{bmatrix}\begin{bmatrix}k_{1} \\k_{2} \\k_{3} \\k_{4}\end{bmatrix}}} & (15)\end{matrix}$

Here, an angle θi before distortion transformation and an angle Edi(where I=1, 2, . . . , n) thereafter are swapped one for another so asto redefine the above expression as a transformation matrix [k_(inv)(1)k_(inv)(2) k_(inv)(3) k_(inv)(4)] for distortion correction of [k1 k2 k3k4]T. Thus, having the left side of the expression multiplied by apseudo inverse matrix of the (n x 4) matrix on the right side enablesestimation of the parameter k_(inv) as defined by the followingexpression (16).

[Math. 16]

k _(inv)=[k _(inv(1)) k _(inv(2)) k _(inv(3)) k _(inv(4))]^(T)  (16)

The points (pixels) for use in the estimation are to be a sufficientnumber of points sampled at equal intervals longitudinally and crosswiseover the entire image. The lens distortion factor k in the direction ofthe distorted two-dimensional image (in the direction of re-projection)is to be given an appropriate initial value, such as a representativevalue generated from an average of the calibration values of multipleprojection apparatuses 112 (projectors) and imaging apparatuses 113(cameras).

Further, correcting the lens distortion using the parameter k_(inv)requires tracking inversely the above-described distortion processing.Given the coordinates x′,y′ including the distortion, adistortion-corrected angle of view θ′ is obtained as defined by thefollowing expressions (17) and (18).

[Math. 17]

x′ ² =x′ ² +y′ ²  (17)

[Math. 18]

θ′=r′(1+k _(inv(1)) r′ ² +k _(inv(2)) r′ ⁴ +k _(inv(3)) r′ ⁶ k _(inv(4))r′ ⁸)  (18)

Then the homogeneous coordinates a,b corresponding to the angle of viewθ′ are obtained as defined by the following expressions (19) and (20).

[Math. 19]

a=(tan θ′/r′)x′  (19)

[Math. 20]

b=(tan θ′/r′)y′  (20)

This makes it possible to correct the coordinates including thedistortion into the distortion-free coordinates.

<Flow of the Parameter Estimation Process>

In steps S121 and S122 of FIG. 8, the parameter k_(inv) corresponding toinverse transformation of the lens distortion factor is re-estimated.The re-estimation is implemented by nonlinear optimization that involvesminimizing the distance between corresponding light rays (triangulationerror) obtained from the pixel-to-pixel correspondence between theprojection apparatus 112 and the imaging apparatus 113. A typical flowof the parameter estimation process constituted by steps S121 and S122in FIG. 8 is explained below with reference to the flowchart of FIG. 9.As the algorithm for nonlinear optimization, the Levenberg-Marquardtmethod may be used, for example.

When the parameter estimation process is started, the imaging variableestimation section 261 (or projection variable estimation section 262)sets, for example, the parameter k_(inv) as an estimation targetparameter in step S141. Here, any internal or external variable otherthan the parameter k_(inv) may be designated instead. Further, multipleparameters may be designated at the same time.

In step S142, the imaging variable estimation section 261 (or projectionvariable estimation section 262) performs distortion correction on eachcorresponding point by using the parameter k_(inv) corresponding toinverse transformation of the lens distortion factor.

With the present method, as discussed above, the three-dimensional spaceis regarded as a distortion-free ideal space, and the captured orprojected image is defined to include distortion caused by the fθ lens.That is, previously acquired pixel-to-pixel correspondence between theprojection apparatus 112 and the imaging apparatus 113 has been obtainedas the corresponding relations between the image coordinates of pixelseach including distortion. Thus, in order to perform a ray trace in thethree-dimensional space on the basis of the corresponding relations, itis necessary to use distortion-free image coordinates.

In view of the above, a corresponding pixel 601 in an image 611 on atwo-dimensional plane such as the one depicted on the left in FIG. 10 issubjected to distortion correction using the parameter k_(inv) and theabove-described expressions (7) to (11). An image coordinate value(corresponding pixel 602) is thus obtained in a distortion-free image612 whose range is made wider than the initial rectangle due to theeffect of the fθ lens, as depicted on the right in FIG. 10. Thisdistortion correction is carried out on each of the correspondingpixels.

In step S143, the imaging variable estimation section 261 (or projectionvariable estimation section 262) calculates (approximates) triangulationpoints by tracing corresponding light rays. The light rays correspondingto a pixel can be traced by projecting distortion-free image coordinatesin the direction of the three-dimensional space, by using the relationsof the expressions (10) and (11) above. That is, as depicted in FIG. 11,the distortion correction transforms a corresponding pixel 601A in adistorted two-dimensional plane image 611A to a corresponding pixel 602Ain an image 612A. Likewise, the distortion correction transforms acorresponding pixel 601B in a distorted two-dimensional plane image 611Bto a corresponding pixel 602B in an image 612B. In this manner, thecorresponding pixels between the projection apparatus 112 and theimaging apparatus 113 are corrected for distortion and subjected toprojection to obtain light rays of which the intersection point ismeasured as the three-dimensional point corresponding to the pixel.

At this time, the distance between the light rays corresponding to atriangulation point is regarded as an error between the correspondinglight rays (i.e., triangulation error). In the case where the error iszero, that means the corresponding light rays intersect with each otherat one point in the three-dimensional space. After the three-dimensionalcoordinates of each corresponding point (also referred to ascorresponding three-dimensional coordinates) and the concomitantmeasurement error have been obtained in this manner, the processadvances to step S144.

In step S144, the imaging variable estimation section 261 (or projectionvariable estimation section 262) calculates an average error of theentire corresponding points on the basis of the measurement errorsobtained in the processing of step S143.

In step S145, the imaging variable estimation section 261 (or projectionvariable estimation section 262) determines whether or not allcorresponding points have been processed. In the case where anunprocessed corresponding point is determined to exist, the processreturns to step S141 and the subsequent steps are repeated. That is,steps S141 to S145 are carried out on each of the corresponding points.Then, in the case where it is determined in step S145 that allcorresponding points have been processed, the process advances to stepS146.

In step S146, the imaging variable estimation section 261 (or projectionvariable estimation section 262) determines whether or not the averageerror calculated in step S144 is equal to or smaller than apredetermined threshold value. In the case where the average error isdetermined to be larger than the predetermined threshold value, theprocess advances to step S147.

In step S147, the imaging variable estimation section 261 (or projectionvariable estimation section 262) corrects the estimation parameters(e.g., parameter k_(inv) and other parameters). Note that in order toachieve highly accurate estimation, the corresponding points of whichthe errors are very large are to be removed as needed by each processingblock so that these points will not be used for the estimation. Uponcompletion of the processing in step S147, the process returns to stepS141 and the subsequent steps are repeated. That is, the processing instep S141 to step S147 is repeated until the average error of the entirecorresponding points is optimized to be equal to or smaller than thepredetermined threshold value.

Further, in the case where it is determined in step S146 that theaverage error is equal to or smaller than the predetermined thresholdvalue and that the parameters have been optimized, the parameterestimation process is terminated. The process then returns to theflowchart of FIG. 8.

That is, outliers are removed so that the points outside the screen 121or the points of low sensing accuracy will not be used for theestimation. For example, as depicted in the upper part of FIG. 12, anerror between corresponding light rays from the projection apparatus 112and imaging apparatus 113 is obtained at the time of finding atriangulation point. Then, as illustrated in the lower part of FIG. 12,the optimization for minimizing the error between the correspondinglight rays and the removal of the sensing points with very large errorsbetween the corresponding light rays are repeated. This enables stillmore accurate estimation of the internal and external variables in thecase where highly accurate information regarding the correspondingpoints has been obtained from the sensing process.

Note that in the total optimization in step S123 of FIG. 8, theparameters are optimized basically in a similar manner. It is to benoted, however, that during the total optimization, the process ofupdating all projection and imaging variables estimated as describedabove is repeated while the parameters targeted for the estimation arechanged one after another.

When the posture estimation process and the parameter estimation processare performed as described above, it is possible to achieve highlyaccurate, automated projection correction by precisely estimating thelens distortion factor during automatic calibration of the projectionimaging system 100 that includes the projection apparatuses 112 andimaging apparatuses 113 each using the fish-eye lens (fθ lens) highlysusceptible to the effect of lens distortion.

<Flow of the Geometric Correction Process>

Explained next with reference to the flowchart of FIG. 13 is a typicalflow of the geometric correction process performed in step S103 of FIG.6.

When the geometric correction process is started, the projection planemodeling section 271 in the geometric correction section 253reconfigures in step S161 the projection plane by using the parametersrelated to posture estimation of the projection apparatuses 112 andimaging apparatuses 113, thereby fitting a two-dimensional curvedsurface.

In step S162, the virtual viewpoint position/projection directionestimation section 272 estimates a virtual viewpoint position and adirection of image projection relative to that virtual viewpointposition. For example, suppose that the processing in step S161 has seta screen shape model 701 as illustrated in FIG. 14. Then, the virtualviewpoint position/projection direction estimation section 272 sets avirtual viewpoint 702 at the front of the screen shape model 701 andestablishes a projection direction (in front) relative to the virtualviewpoint 702.

In order to project an image from the virtual viewpoint 702 to thescreen shape model 701 in a geometrically accurate manner, it isnecessary to determine the front, horizontal, and vertical directions ofthe virtual viewpoint 702. Thus, to determine the front direction, thevirtual viewpoint position/projection direction estimation section 272selects, from a group of three-dimensional points measured as depictedin subfigure A of FIG. 15, a group of points corresponding to the edgeof the screen 121 as illustrated in subfigure B of FIG. 15. The virtualviewpoint position/projection direction estimation section 272 furtherfits the selected group of points to a plane as pictured in subfigure Cof FIG. 15. A normal direction to that plane is regarded as the frontdirection as viewed from a viewpoint camera. Further, because theprojection apparatuses 112-1 and 112-2 are generally arrangedapproximately at the same height, the horizontal direction is determinedon the basis of that height of the projection apparatuses 112. Thevertical direction is finally determined as a vector that is at rightangles to the other two directions. Such estimation of the virtualviewpoint direction and of the projection direction is automaticallyperformed using measurement information. There is no need to designatethe direction explicitly and manually.

The above processing allows the virtual viewpoint direction and theprojection direction to be estimated. Thus, on the assumption that thescreen 121 is positioned in front of the virtual viewpoint position, ageometric correction vector can be generated in such a manner as topermit viewing of the image from that position in a geometricallyaccurate manner.

In step S163, the model misalignment corresponding processing section273 determines whether or not the estimates correspond to modelmisalignment. In the case where it is determined that there ismisalignment between the projection plane and its model (modelmisalignment) and that the estimates correspond to model misalignment,the process advances to step S164.

In step S164, the model misalignment corresponding processing section273 performs a model misalignment corresponding interpolation process.In the reconfiguration of the projection plane in step S161, as depictedin subfigure A of FIG. 16, three-dimensional points 721 on theprojection plane are measured using the information regarding thecorresponding points between the projection apparatus 112 and theimaging apparatus 113 and the internal and external variable estimatesof these apparatuses. Then, as illustrated in subfigure B of FIG. 16,this group of measured three-dimensional points is fitted with thescreen shape model 701 of a two-dimensional curved surface (e.g.,ellipsoid, hyperboloid, or cylinder) such as to minimize theleast-square error. In the case of a spherical surface screen, thepoints are usually modeled as an ellipsoid. This provides smooth,noise-resistant geometric correction through calculation of thegeometric correction vector based on the model estimated from the wholepixels.

It is to be noted, however, that the model-based geometric correctioncan entail an error of the screen shape when the shape is measured asthree-dimensional points deviating, due to distortion, for example, fromthe model estimated as depicted in FIG. 17. In the example of FIG. 17, athree-dimensional point 731 measured on the estimated screen shape model701 is found deviating from a three-dimensional point 732 that ought tobe measured on the actual (real-world) screen 121.

Thus, as illustrated in FIG. 18, the geometric correction vector isgenerated from the position of an intersection point on a model with itsradius set by the sphere center of a virtually arranged spherical modeland an interpolated value of distances of triangulation points. That is,in the example of FIG. 18, a three-dimensional point 741 on a screenshape model 701A and a three-dimensional point 742 on a screen shapemodel 701B are measured as the triangulation points. A three-dimensionalpoint 743 is then interpolated between these three-dimensional points(i.e., an assumed intersection point on a screen shape model 701C withits radius interpolated relative to the sphere center). Thethree-dimensional point 743 is then used to generate the geometriccorrection vector.

In this manner, it is possible to implement smooth correction even onthose model misalignment portions of the actual screen 121 that are notaligned with the ideal model (i.e., screen shape model 701).

Then, in step S165, the model misalignment corresponding processingsection 273 generates a correction vector corresponding to the modelmisalignment (also called the model misalignment correspondingcorrection vector). Upon completion of the processing in step S165, theprocess advances to step S167.

In addition, in the case where it is determined in step S163 that theestimates do not correspond to model misalignment, the process advancesto step S166. In step S166, the model misalignment correspondingprocessing section 273 generates a correction vector not correspondingto the model misalignment (also called the model misalignmentnon-corresponding correction vector). Upon completion of the processingin step S166, the process advances to step S167.

In step S167, the projection mask generation section 274 generates aprojection mask such as to limit the range of image projection to anarea inside the screen 121 (i.e., the projected image does not protrudefrom the screen 121 (confined within the screen 121)). The projectionmask generation section 274 outputs geometric correction vector dataincluding the correction vector generated in step S165 or S166 and theprojection mask generated in step S167. Upon completion of theprocessing in step S167, the geometric correction process is terminated.The process then returns to the flowchart of FIG. 6.

By performing the above processes as described, the control apparatus111 implements correction of projection to the dome-type screen 121 foruse with multiple projection apparatuses 112 each using the fθ lens, bymeans of a three-dimensional approach using the projection imagingsystem 100.

In this case, images are corrected on the basis of a two-dimensionalcurved surface model built on three-dimensional informationrepresentative of a screen shape. The processing thus guarantees thegeometric accuracy of images, such as a straight line being seen as acorrect straight line. Further, in the projection imaging system 100having a fish-eye (equidistant projection) optical assembly, theprocessing easily enables the steps ranging from calibration of theinternal and external variables of the projection apparatuses 112 andimaging apparatuses 113 to projection correction, the steps not havingbeen dealt with successfully by the methods involving the use of aprojector-camera system based on the existing perspective projectionoptical assembly.

Also, it is not necessary to perform preliminary estimation (preliminarycalibration) of part or all of the internal and external variables ofeach of the projection and imaging apparatuses, as opposed to thethree-dimensional approach involving ordinary projection imaging systemsrequiring such preliminary estimation.

Moreover, geometrically accurate correction is implemented with respectto an established virtual viewpoint without the imaging apparatus 113being arranged in the position desired to be the viewpoint. Further,from the virtual viewpoint, projection is performed to the screen 112 inan appropriate projection direction with suitable up-down/left-rightimage orientation maintained. Further, the projected image is correctedin a geometrically accurate manner relative to the front of the screen121 on the basis of the information obtained with projection and imagingby the projection apparatus 112 and imaging apparatus 113 not in frontof the screen 121 but from the side thereof.

Also, as described above, the parameter k_(inv) corresponding to inversetransformation of the lens distortion factor of the projectionapparatuses 112 is introduced in parameter estimation, the parameterk_(inv) being used to uniformly correct the pixels for distortion. Thiseliminates the need for compensating each of the pixels with use of thelens distortion factor k.

Further, ray traces are performed by correcting the pixels of adistorted image for distortion and by projecting the pixels onto adistortion-free image. Thus, the optimization for minimizing thedistance between corresponding light rays achieves re-estimation of theinternal and external variables of the projection apparatuses 112 andimaging apparatuses 113 including the parameter k_(inv).

In addition, some of the internal parameters of the projectionapparatuses 112 and imaging apparatuses 113 need only be givensufficiently appropriate initial values (e.g., averages of calibrationvalues). This still provides calibration of the whole internal andexternal variables of the projection apparatuses 112 and imagingapparatuses 113 that are arranged before the screen without recourse topreliminary calibration procedures for each of the apparatus enclosures.

It has been explained above that the control apparatus 111 estimates theinternal and external variables of each of the projection apparatuses112 and imaging apparatuses 113. However, the present method is notlimited to the foregoing examples. Alternatively, the control apparatus111 may estimate the internal and external variables of either theprojection apparatuses 112 or the imaging apparatuses 113. In such acase, the internal and external variables of the remaining apparatusesmay be determined beforehand or may be estimated by the otherapparatuses.

As another alternative, the control apparatus 111 may estimate eitherthe internal variables or the external variables. In such a case, theother variables may be determined beforehand or may be estimated by theother apparatuses.

That is, the control apparatus 111 need only estimate at least eitherthe internal variables or the external variables of either theprojection apparatuses 112 or the imaging apparatuses 113.

3. Second Embodiment

<Other Configurations>

Note that the configuration of the imaging system to which the presenttechnology is applied is not limited to the above-described example inFIG. 1. For example, the control apparatus 111, the projection apparatus112, and the imaging apparatus 113 may be provided in desired numbers.The control apparatus 111 may, for example, be provided in pluralnumbers. There may be three or fewer, or five or more projectionapparatuses 112 and three or fewer, or five or more imaging apparatuses113. Also, the projection apparatus 112 and the imaging apparatus 113need not be provided in equal numbers.

Further, it has been explained above that the projection apparatuses 112and the imaging apparatuses 113 are connected with the control apparatus111 via the cables 115. Alternatively, these apparatuses may beinterconnected in any other suitable manner as long as they cancommunicate with each other. For example, the control apparatus 111 maycommunicate with the projection apparatuses 112 and imaging apparatuses113 by wire or wirelessly, or in both wired and wireless fashion. Asanother alternative, the control apparatus 111, the projectionapparatuses 112, and the imaging apparatuses 113 may be interconnectedcommunicably via any suitable communication network.

Any suitable method of communication may be adopted for that network.For example, the method may be wired communication or wirelesscommunication, or both. The network may include a single or multiplecommunication networks. For example, this network may include theInternet, public telephone networks, mobile broadband networks such aswhat is generally called the 3G or 4G networks, WAN (Wide Area Network),LAN (Local Area Network), wireless communication networks forcommunication based on the Bluetooth (registered trademark) standard,communication channels for short-range wireless communication such asNFC (Near Field Communication), communication channels for infrared raycommunication, wired communication networks based on such standards asHDMI (High-Definition Multimedia Interface; registered trademark) andUSB (Universal Serial Bus), or any other communication networks andcommunication channels based on any suitable communication standards.

Also, the projection apparatus 112 and the imaging apparatus 113 may beintegrated into a single apparatus. For example, as depicted insubfigure A of FIG. 19, the projection imaging system 100 may include aprojection imaging apparatus 801-1, a projection imaging apparatus801-2, and a control apparatus 111.

The projection imaging apparatus 801-1 includes a projection section811-1 and an imaging section 812-1. The projection section 811-1 hasfunctions similar to those of the projection apparatus 112-1 in FIG. 1.Also, the imaging section 812-1 has functions similar to those of theimaging apparatus 113-1 in FIG. 1. That is, the projection imagingapparatus 801-1 has the functions of the projection apparatus 112-1 andimaging apparatus 113-1.

Likewise, the projection imaging apparatus 801-2 includes a projectionsection 811-2 and an imaging section 812-2. The projection section 811-2has functions similar to those of the projection apparatus 112-2 inFIG. 1. Also, the imaging section 812-2 has functions similar to thoseof the imaging apparatus 113-2 in FIG. 1. That is, the projectionimaging apparatus 801-2 has the functions of the projection apparatus112-2 and imaging apparatus 113-2.

The control apparatus 111 is communicably connected with the projectionimaging apparatus 801-1 via the cable 115-1. The communication allowsthe control apparatus 111 to control the projection imaging apparatus801-1. Thus controlled, the projection imaging apparatus 801-1 issupplied with an image, projects the image to the projection plane, andcaptures a projected image on the projection plane for imageacquisition. The control apparatus 111 is also connected communicablywith the projection imaging apparatus 801-2 via the cable 115-2. Thecommunication allows the control apparatus 111 to control the projectionimaging apparatus 801-2. Thus controlled, the projection imagingapparatus 801-2 is supplied with an image, projects the image to theprojection plane (e.g., screen 121), and captures a projected image onthe projection plane for image acquisition.

That is, in this case as well, the projection imaging system 100 canperform image projection correction by use of the present technology andin a manner similar to the case in FIG. 1.

Note that the projection imaging apparatuses 801-1 and 801-2 arereferred to as the projection imaging apparatus 801 in the case wherethere is no need for their individual explanation. Also, the projectionsections 811-1 and 811-2 are referred to as the projection section 811where there is no need for their individual explanation. Further, theimaging sections 812-1 and 812-2 are referred to as the imaging section812 where there is no need for their individual explanation.

As in the case in FIG. 1, the projection imaging apparatus 801 may beprovided in desired numbers. For example, three or more projectionimaging apparatuses 801 may be provided. Further, there may be providedone or more projection sections 811 and one or more imaging sections 812in the projection imaging apparatus 801; the projection section 811 andthe imaging section 812 may be provided in different numbers. Moreover,each of the projection imaging apparatuses 801 may include differentnumbers of the projection sections 811 and imaging sections 812.Further, the projection imaging system 100 may include the projectionimaging apparatus 801 and either the projection apparatus 112 or theimaging apparatus 113, or the projection imaging apparatus 801 and boththe projection apparatus 112 and the imaging apparatus 113 in a mixedmanner.

Further, the control apparatus 111 may be integrated with anotherapparatus. For example, as depicted in subfigure B of FIG. 19, theprojection imaging system 100 may include an imaging apparatus 820, aprojection apparatus 112, and a projection imaging apparatus 801.

The imaging apparatus 820 includes the imaging section 812-1 and acontrol section 821. The control section 821 has functions similar tothose of the control apparatus 111 in FIG. 1 or in subfigure A of FIG.19. That is, the imaging apparatus 820 has the functions of the imagingapparatus 113 and the control apparatus 111. The imaging apparatus 820,the projection apparatus 112, and the projection imaging apparatus 801are connected communicably with each other via the cable 115.

Images are thus supplied to the imaging apparatus 820 via the cable 114.The control section 821 in the imaging apparatus 820 controls, via thecable 115, the projection section 811-1 in the projection apparatus 112and the projection section 811-2 in the projection imaging apparatus 801to project the supplied image to the projection plane (e.g., screen121). The control section 821 further controls the imaging section 812-1as well as the imaging section 812-2 in the projection imaging apparatus801 via the cable 115 to capture the projected image on the projectionplane. At this time, the control section 821 performs geometriccorrection on the image by using the present technology so that ageometrically corrected image will be projected.

That is, also in this case, the projection imaging system 100 canperform image projection correction by use of the present technology andin a manner similar to the case in FIG. 1.

Note that the control apparatus 111 may be integrated with an apparatusother than the imaging apparatus 113, such as with the projectionapparatus 112 or the projection imaging apparatus 801. That is, theapparatus that includes the control section 821 may be configured in anysuitable manner and may have the projection sections 811 and imagingsections 812 in desired numbers. Also, there may be one or multipleapparatuses each having the control section 821. Further, in theprojection imaging system 100, the constituent elements other than theapparatus having the control section 821 may be configured as desired.The configuration of the projection imaging system 100 is thus notlimited to that of the example in subfigure B of FIG. 19.

Furthermore, the entire configuration of the projection imaging system100 may be integrated into a single apparatus. For example, as depictedin subfigure C of FIG. 19, the whole system may be integrated into aprojection imaging apparatus 830. In the example in subfigure C of FIG.19, the projection imaging apparatus 830 includes a projection section811-1, a projection section 811-2, an imaging section 812-1, an imagingsection 812-2, and a control section 821. That is, the projectionimaging apparatus 830 is configured in a manner similar to theprojection imaging system 100 in FIG. 1. As described above, the presenttechnology may be applied internally to the projection imaging apparatus830.

Obviously, the projection imaging apparatus 830 may be configured asdesired and the configuration is not limited to that of the example insubfigure C of FIG. 19. For example, the control section 821, theprojection section 811, and the imaging section 812 may each be providedin desired numbers.

4. Notes

<Software>

The series of the processes described above may be executed either byhardware or by software. In the case where these processes are to becarried out by software, the programs constituting the software areinstalled from a network or from a recording medium.

In the case of the control apparatus 111 in FIG. 2, for example, itsrecording medium is constituted by the removable media 221 on which theprograms are recorded and which are distributed to users apart from theapparatus in order to deliver the recorded programs. In such a case, forexample, a piece of the removable media 221 on which the programs arerecorded may be attached to the drive 215 so as to have the programsinstalled into the storage section 213 following their retrieval fromthe attached piece of removable media 221.

As another example, in the case of the projection apparatus 112 in FIG.4, its recording medium is constituted by the removable media 321 onwhich the programs are recorded and which are distributed to users apartfrom the apparatus in order to deliver the recorded programs. In such acase, for example, a piece of the removable media 321 on which theprograms are recorded may be attached to the drive 315 so as to have theprograms installed into the storage section 313 following theirretrieval from the attached piece of the removable media 321.

As a further example, in the case of the imaging apparatus 113 in FIG.5, its recording medium is constituted by the removable media 421 onwhich the programs are recorded and which are distributed to users apartfrom the apparatus in order to deliver the recorded programs. In such acase, for example, a piece of the removable media 421 on which theprograms are recorded may be attached to the drive 415 so as to have theprograms installed into the storage section 413 following theirretrieval from the attached piece of removable media 421.

Further, the programs may be offered via wired or wireless transmissionmedia such as local area networks, the Internet, and digital satellitebroadcasts. In the case of the control apparatus 111 in FIG. 2, forexample, the programs may be received by the communication section 214and installed into the storage section 213. Further, in the case of theprojection apparatus 112 in FIG. 4, for example, the programs may bereceived by the communication section 314 and installed into the storagesection 313. Further, in the case of the imaging apparatus 113 in FIG.5, for example, the programs may be received by the communicationsection 414 and installed into the storage section 413.

Otherwise, the programs may be preinstalled in a storage section or aROM. In the case of the control apparatus 111 in FIG. 2, for example,the programs may be preinstalled in the storage section 213 or in a ROM(not depicted) inside the control section 201. Further, in the case ofthe projection apparatus 112 in FIG. 4, for example, the programs may bepreinstalled in the storage section 313 or in a ROM (not depicted)inside the control section 301. Further, in the case of the imagingapparatus 113 in FIG. 5, for example, the programs may be preinstalledin the storage section 413 or in a ROM (not depicted) inside the controlsection 401.

<Targets to which the Present Technology May be Applied>

Further, the present technology may be implemented as any of thecomponents constituting an apparatus or any of the apparatusesconfiguring a system, such as a processor (e.g., video processor) in theform of a system LSI (Large Scale Integration), a module (e.g., videomodule) using multiple processors, a unit (e.g., video unit) usingmultiple modules, and a set (e.g., video set) supplementing a unit withother functions (i.e., as part of the apparatus).

Furthermore, the present technology may also be applied to a networksystem including multiple apparatuses. For example, the technology maybe applied to cloud services that offer image-related (video-related)services to any types of terminals such as computers, AV (Audio Visual)equipment, mobile information processing terminals, and IoT (Internet ofThings) devices.

Note that the systems, apparatuses, or processing sections to which thepresent technology is applied may be used for desired purposes in anytypes of fields such as transportation, healthcare, crime prevention,agriculture, livestock farming, mining, beauty care, factories, homeelectric appliances, climate, and nature monitoring.

For example, the present technology may be applied to systems anddevices used for offering content for aesthetic or appreciativepurposes. As another example, the present technology may be applied tosystems and devices for transportation-related purposes such as formonitoring traffic conditions and controlling automated driving. As afurther example, the present technology may be applied to systems anddevices for security purposes. As an even further example, the presenttechnology may be applied to systems and devices for automated controlof machines. As a still further example, the present technology may beapplied to systems and devices for use in agriculture and livestockfarming. As a yet further example, the present technology may be appliedto systems and devices for monitoring the state of nature such asvolcanoes, forests and oceans, as well as the state of wildlife. Asanother example, the present technology may be applied to systems anddevices for use in sports.

<Others>

The present technology is not limited to the embodiments discussed aboveand may be implemented in diverse variations so far as they are withinthe scope of the appended claims or the equivalents thereof.

For example, the present technology may be implemented as any of thecomponents constituting an apparatus or a system, such as a processor(e.g., video processor) in the form of a system LSI (Large ScaleIntegration), a module (e.g., video module) using multiple processors, aunit (e.g., video unit) using multiple modules, and a set (e.g., videoset) supplementing a unit with other functions (i.e., as part of theapparatus).

Note that in this description, the term “system” refers to an aggregateof multiple components (e.g., apparatuses or modules (parts)). It doesnot matter whether all components are housed in the same enclosure.Thus, a system may be configured with multiple apparatuses housed inseparate enclosures and interconnected via a network, or with a singleapparatus in a single enclosure that houses multiple modules.

Further, for example, any configuration explained in the foregoingparagraphs as one apparatus (or processing section) may be divided intomultiple apparatuses (or processing sections). Conversely, theconfigurations explained above as multiple apparatuses (or processingsections) may be unified into one apparatus (or processing section).Also, the configuration of each apparatus (or processing section) mayobviously be supplemented with a configuration or configurations otherthan those discussed above. Furthermore, part of the configuration of anapparatus (or processing section) may be included in the configurationof another apparatus (or processing section), provided theconfigurations and the workings remain substantially the same for thesystem as a whole.

As another example, the present technology may be implemented as a cloudcomputing setup in which a single function is processed cooperatively bymultiple networked apparatuses on a shared basis.

As another example, the above-described programs may be executed by anyapparatus. In such a case, the apparatus is only required to havenecessary functions (e.g., functional blocks) and obtain necessaryinformation for program execution.

Also, each of the steps discussed in reference to the above-describedflowcharts may be executed either by a single apparatus or by multipleapparatuses on a shared basis. Further, if a single step includesmultiple processes, these processes may be executed either by a singleapparatus or by multiple apparatuses on a shared basis. In other words,multiple steps included in a single step may be executed as a process ofmultiple steps. Conversely, the process explained as made up of multiplesteps may be executed as a single step.

Note that the programs executed by the computer may each be processed insuch a manner that the processes of the steps describing the program arecarried out chronologically, i.e., in the sequence depicted in thisdescription, in parallel with other programs, or in otherwiseappropriately timed fashion such as when the program is invoked asneeded. That is, the above processes of steps may be carried out insequences different from those discussed above as long as there is noconflict between the steps. Furthermore, the processes of the stepsdescribing a given program may be performed in parallel with, or incombination with, the processes of other programs.

Note that the multiple techniques discussed in the present descriptionmay each be implemented independently of the others as long as there isno inconsistency therebetween. Obviously, any number of these techniquesmay be implemented in combination. For example, some or all of thetechniques discussed in conjunction with one embodiment may beimplemented in combination with some or all of the techniques explainedin connection with another embodiment. Further, some or all of any ofthe techniques discussed above may be implemented in combination withanother technique not described above.

Note that the advantageous effects stated in the present description areonly examples and not limitative of the present technology that may alsoprovide other advantageous effects.

Note that the present disclosure may also be implemented preferably inthe following configurations:

(1)

An information processing apparatus including:

a posture estimation section configured such that, by use of an imageprojection model using a distortion factor of an fθ lens with an imageheight of incident light expressed by a product of a focal pointdistance f and an incident angle θ of the incident light, the postureestimation section estimates a posture of a projection section forprojecting an image and a posture of an imaging section for capturing aprojection plane to which the image is projected.

(2)

The information processing apparatus as stated in paragraph (1) above inwhich,

by use of the image projection model, the posture estimation sectionestimates posture-related parameters of at least either the projectionsection or the imaging section.

(3)

The information processing apparatus as stated in paragraph (2) above,in which

the posture-related parameters include internal parameters of at leasteither the projection section or the imaging section.

(4)

The information processing apparatus as stated in paragraph (3) above,in which

the internal parameters include at least one of a focal point distance,a principal point, and a parameter corresponding to inversetransformation of the distortion factor regarding either the projectionsection or the imaging section.

(5)

The information processing apparatus as stated in any one of paragraphs(2) to (4) above, in which

the posture-related parameters include external parameters of at leasteither the projection section or the imaging section.

(6)

The information processing apparatus as stated in paragraph (5) above,in which

the external parameters include either a rotation matrix or atranslation vector with respect to an origin of a world coordinatesystem of either the projection section or the imaging section.

(7)

The information processing apparatus as stated in any one of paragraphs(2) to (6) above, in which

the posture estimation section

-   -   performs image distortion correction on the projection section        and the imaging section by using the parameter corresponding to        inverse transformation of the distortion factor, and    -   performs a ray trace to detect a corresponding point by use of        the projection section and the imaging section subjected to the        distortion correction, thereby estimating the posture-related        parameters.        (8)

The information processing apparatus as stated in paragraph (7) above,in which

the posture estimation section optimizes the posture-related parametersin such a manner that an average error of the detected correspondingpoints becomes equal to or smaller than a predetermined threshold value.

(9)

The information processing apparatus as stated in paragraph (8) above inwhich,

in a case where the average error does not become equal to or smallerthan the threshold value, the posture estimation section corrects theparameters for use in estimation of the posture-related parameters, and

the posture estimation section repeatedly estimates the posture-relatedparameters until the average error becomes equal to or smaller than thethreshold value.

(10)

The information processing apparatus as stated in paragraph (8) or (9)above, in which

the posture estimation section optimizes the posture-related parameterswhile removing as an outlier a corresponding point having a large error.

(11)

The information processing apparatus as stated in any one of paragraphs(7) to (10) above, in which

the posture estimation section

-   -   estimates the posture-related parameters of the projection        section,    -   estimates the posture-related parameters of the imaging section,        and    -   optimizes the estimated posture-related parameters of the        projection section and the estimated posture-related parameters        of the imaging section.        (12)

The information processing apparatus as stated in any one of paragraphs(2) to (11) above, further including:

a geometric correction section configured such that, by use of theposture-related parameters estimated by the posture estimation section,the geometric correction section generates vector data for geometriccorrection of the image projected by the projection section.

(13)

The information processing apparatus as stated in paragraph (12) above,in which

the geometric correction section

-   -   obtains the projection plane by use of the posture-related        parameters of the projection section and of the imaging section        so as to model the obtained projection plane as a        two-dimensional curved surface, and    -   generates the vector data by use of the projection plane model        thus obtained.        (14)

The information processing apparatus as stated in paragraph (13) above,in which,

by use of the projection plane model, the geometric correction sectionestimates a virtual viewpoint position in front of the projection planeand a projection direction relative to the virtual viewpoint position,thereby generating the vector data for suppressing distortion of thevirtual viewpoint position.

(15)

The information processing apparatus as stated in paragraph (14) above,in which

the geometric correction section performs a model misalignmentcorresponding process for suppressing an error between an actualprojection plane and the model.

(16)

The information processing apparatus as stated in paragraph (15) above,in which

the geometric correction section generates a projection mask forlimiting a range in which the image is to be projected.

(17)

The information processing apparatus as stated in any one of paragraphs(1) to (16) above, further including:

a corresponding point detection section configured to detect acorresponding point between the projection section and the imagingsection, in which,

by use of the corresponding points detected by the corresponding pointdetection section, the posture estimation section estimates the postureof the projection section and that of the imaging section.

(18)

The information processing apparatus as stated in any one of paragraphs(1) to (17) above, further including:

the projection section.

(19)

The information processing apparatus as stated in any one of paragraphs(1) to (18) above, further including:

the imaging section.

(20)

An information processing method including:

by use of an image projection model using a distortion factor of an fθlens with an image height of incident light expressed by a product of afocal point distance f and an incident angle θ of the incident light,estimating a posture of a projection section for projecting an image anda posture of an imaging section for capturing a projection plane towhich the image is projected.

REFERENCE SIGNS LIST

100 Projection imaging system, 111 Control apparatus, 112 Projectionapparatus, 113 Imaging apparatus, 201 Control section, 251 Sensingprocessing section, 252 Posture estimation section, 253 Geometriccorrection section, 261 Imaging variable estimation section, 262Projection variable estimation section, 263 Total optimization section,271 Projection plane modeling section, 272 Virtual viewpointposition/projection direction estimation section, 273 Model misalignmentcorresponding processing section, 274 Projection mask generationsection, 301 Control section, 302 Projection section, 401 Controlsection, 402 Imaging section, 801 Projection imaging apparatus, 811Projection section, 812 Imaging section, 820 Imaging apparatus, 821Control section, 830 Projection imaging apparatus

1. An information processing apparatus comprising: a posture estimationsection configured such that, by use of an image projection model usinga distortion factor of an fθ lens with an image height of incident lightexpressed by a product of a focal point distance f and an incident angleθ of the incident light, the posture estimation section estimates aposture of a projection section for projecting an image and a posture ofan imaging section for capturing a projection plane to which the imageis projected.
 2. The information processing apparatus according to claim1, wherein, by use of the image projection model, the posture estimationsection estimates posture-related parameters of at least either theprojection section or the imaging section.
 3. The information processingapparatus according to claim 2, wherein the posture-related parametersinclude internal parameters of at least either the projection section orthe imaging section.
 4. The information processing apparatus accordingto claim 3, wherein the internal parameters include at least one of afocal point distance, a principal point, and a parameter correspondingto inverse transformation of the distortion factor regarding either theprojection section or the imaging section.
 5. The information processingapparatus according to claim 2, wherein the posture-related parametersinclude external parameters of at least either the projection section orthe imaging section.
 6. The information processing apparatus accordingto claim 5, wherein the external parameters include either a rotationmatrix or a translation vector with respect to an origin of a worldcoordinate system of either the projection section or the imagingsection.
 7. The information processing apparatus according to claim 2,wherein the posture estimation section performs image distortioncorrection on the projection section and the imaging section by using aparameter corresponding to inverse transformation of the distortionfactor, and performs a ray trace to detect a corresponding point by useof the projection section and the imaging section subjected to thedistortion correction, thereby estimating the posture-relatedparameters.
 8. The information processing apparatus according to claim7, wherein the posture estimation section optimizes the posture-relatedparameters in such a manner that an average error of the detectedcorresponding points becomes equal to or smaller than a predeterminedthreshold value.
 9. The information processing apparatus according toclaim 8, wherein, in a case where the average error does not becomeequal to or smaller than the threshold value, the posture estimationsection corrects the parameters for use in estimation of theposture-related parameters, and the posture estimation sectionrepeatedly estimates the posture-related parameters until the averageerror becomes equal to or smaller than the threshold value.
 10. Theinformation processing apparatus according to claim 8, wherein theposture estimation section optimizes the posture-related parameterswhile removing as an outlier a corresponding point having a large error.11. The information processing apparatus according to claim 7, whereinthe posture estimation section estimates the posture-related parametersof the projection section, estimates the posture-related parameters ofthe imaging section, and optimizes the estimated posture-relatedparameters of the projection section and the estimated posture-relatedparameters of the imaging section.
 12. The information processingapparatus according to claim 2, further comprising: a geometriccorrection section configured such that, by use of the posture-relatedparameters estimated by the posture estimation section, the geometriccorrection section generates vector data for geometric correction of theimage projected by the projection section.
 13. The informationprocessing apparatus according to claim 12, wherein the geometriccorrection section obtains the projection plane by use of theposture-related parameters of the projection section and of the imagingsection so as to model the obtained projection plane as atwo-dimensional curved surface, and generates the vector data by use ofthe projection plane model thus obtained.
 14. The information processingapparatus according to claim 13, wherein, by use of the projection planemodel, the geometric correction section estimates a virtual viewpointposition in front of the projection plane and a projection directionrelative to the virtual viewpoint position, thereby generating thevector data for suppressing distortion of the virtual viewpointposition.
 15. The information processing apparatus according to claim14, wherein the geometric correction section performs a modelmisalignment corresponding process for suppressing an error between theactual projection plane and the model.
 16. The information processingapparatus according to claim 15, wherein the geometric correctionsection generates a projection mask for limiting a range in which theimage is to be projected.
 17. The information processing apparatusaccording to claim 1, further comprising: a corresponding pointdetection section configured to detect a corresponding point between theprojection section and the imaging section, wherein, by use of thecorresponding points detected by the corresponding point detectionsection, the posture estimation section estimates the posture of theprojection section and that of the imaging section.
 18. The informationprocessing apparatus according to claim 1, further comprising: theprojection section.
 19. The information processing apparatus accordingto claim 1, further comprising: the imaging section.
 20. An informationprocessing method comprising: by use of an image projection model usinga distortion factor of an fθ lens with an image height of incident lightexpressed by a product of a focal point distance f and an incident angleθ of the incident light, estimating a posture of a projection sectionfor projecting an image and a posture of an imaging section forcapturing a projection plane to which the image is projected.